NSF PAR Search | NSF Public Access Repository

Note: When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).
What is a DOI Number?

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

BBQ: A hand-built bias benchmark for question answering

https://doi.org/10.18653/v1/2022.findings-acl.165

Parrish, Alicia; Chen, Angelica; Nangia, Nikita; Padmakumar, Vishakh; Phang, Jason; Thompson, Jana; Htut, Phu Mon; Bowman, Samuel (January 2022, Findings of the Association for Computational Linguistics: ACL 2022)

Full Text Available
QuALITY: Question Answering with Long Input Texts, Yes!

Bowman, Samuel R.; Chen, Angelica; He, He; Joshi, Nitish; Ma, Johnny; Nangia, Nikita; Padmakumar, Vishakh; Pang, Richard Yuanzhe; Parrish, Alicia; Phang, Jason; et al (May 2022, NAACL 2022)

To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5,000 tokens, much longer than typical current models can process. Unlike in prior work with passages, our questions are written and validated by contributors who have read the entire passage, rather than relying on summaries or excerpts. In addition, only half of the questions are answerable by annotators working under tight time constraints, indicating that skimming and simple search are not enough to consistently perform well. Our baseline models perform poorly on this task (55.4%) and significantly lag behind human performance (93.5%).
more » « less
Full Text Available
QuALITY: Question Answering with Long Input Texts, Yes!

https://doi.org/10.18653/v1/2022.naacl-main.391

Pang, Richard Yuanzhe; Parrish, Alicia; Joshi, Nitish; Nangia, Nikita; Phang, Jason; Chen, Angelica; Padmakumar, Vishakh; Ma, Johnny; Thompson, Jana; He, He; et al (January 2022, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies)

Full Text Available
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Srivastava, Aarohi; Rastogi, Abhinav; Rao, Abhishek; Shoeb, Abu Awal; Abid, Abubakar; Fisch, Adam; Brown, Adam R.; Santoro, Adam; Gupta, Aditya; Garriga-Alonso, Adri; et al (January 2023, Transactions on machine learning research)

Full Text Available

Search for: All records